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To the P,cesident of the Senate and the 
Speaker of -the House of Repcesentati 




This i^ our report on improvements needed in training 
evaluation: It should help the Congress assess how well 
the, Civil Service Commission and the Federal departments- 
and ageticies are fulfilling the evaluation requirements of 
the Government Employees Training Act, Executive Order 11348 
-and recommendations of a 1967 report by the Subcommittee 
-Ton Manpower and Civil ' Service , House Committee on Post Office 
and Civil Service. 



We made our review pursuant to .the Budget aod Account- 
g Act, 1921 {31 U.S.C. 53), and the Accounting and Audit- 
ng Act. of i950 (31 U.S.C. 67). . ^ 

We are' sending copies of this report to the Director, 
•Office of Management and Budget, to the Chairman, Civil Service 
Commission, and to the Fedejral departments included in our 
review. _ - ' ■ 




Comptroller General 
of the United States 
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,d'i G E S T 

IM> «>iM « 

About 960,000 U.S. civilian emplayiies have- 
received about 45 million hou?:s df »train- ■ 
ing, costing the Federal Govdjirnment approx- . 
imately $216 million. How have the Cj-vil^ , \_ 
Service Commisgion and the Federal Execu- ^ 
tive departments . ; ' 

• —Measured the effectiveness of this . 
training? ^ , 

—Fulfilled the evaluation require- \^ • '. 
. ments of t'he Government Employees. 
Training Act of 4^58? 

V . • . • • / . 

--Progressed in implementing the 
' . recommendations of a 1967 congres- 
^\ sibnal^eufccommittee report which 

concluded that Federal training 
was' not evaluated as required? 

To answer these questions, GAO sent ques- 
tionnaires to about 900 Federal Executive 
department training and employee develop-. , 
ment officers and consulted with Commission ■ 
officials and training' evaluation authorities. 

The questionnaire resu;lts show that .the 
evaluation requirements of the Government 
Employees Training Act, -the Executive order . 
supplementing the law, and the subcommittee 
recommendations are not being met adequately. 

r'The extent ,and degree to. which training 
^ cost data was collected varied widely, 
with many respondents not gathering • 

enough dat^ for effective cos t^ control. . , 

(See p, 10. ) 
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GAO recommends that the Commission define 
and achieve a consensus among the Execu- 
tive d*epartments and affenci-es .on the ele- 
ments that make up the total cost' of 
training and then iyisuve that dat<i on 
these elements is uniformly detevmine,d 
and ,co,ll,ected. 

There was little use of the Goiranission ' s 
Training Cost Model, developed in 1972. ^ 
for making specific cost forecasts, for 
training operations* (Sfee p.-ll.) 

GAO recommends that the Commission deter- 
mine why the cdst"model has been little 
used/ amend and refine it accordingly ^ 
and then increase efforts to publicize it 
to training officia Is at all levels .'^ ^ 

Although almost all respondents reported 
that they had wj^itten training course plans, 
the plans often ^ladsked the, specific fea- 
tures necessary for effective future j:iourse 
evaluation. (See p. 14.) „ c 

GAO recommends that the^„.^€ommission help 
the departments and agencies^ develop 
written training course plans which in- * 
elude instructions for course de livery ^ 
measuring course^ results^' andf modi;^ying 
and updating the course before future use. 




Some perfonnatice^^asures were taken during 
the cOurse,""sf?t{e on its completion, and 
some after the tlra^Qee ' s return to the job. 
The extent, detail, '^'^b^ing, and scope of 
the measures used vafled widely. In many 
cases the use of,x^f^^asur6s was so limited 
m that effective/evaluation was impossible/ 
(See p. n.h 

While many -respondents said that performance 
was assesse4-upon return to the job (prob- • 
ably ^he most criocial step be cause "it 
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demonstrates the adequacy and relevancy 
of the course), .the type, txmmg, and. depth 
of this assessment often fell, short of 
standards suggested by trainirig author- " 
ities. (See p.. 17.) ji 

Statistical and experimental Jvaluations 
weV^ not widely conducted; tipse that Were 
done were elementary. Howevir, oyer 60 
percent of the respondents ^id they were 
satisfied with the level of -evaluations 
performed. (See p. 26.) ; 

Rjespondents' comments indicated that 
problems with training Were widespread 
Long departments, regardless of the sxze 
or scope of the training/<Jfficers '^respon- 
sibilities. 5dost respoi/dents who ^'ere 
dissatisfied with the l^Vel of ^valuations 
lacked resources. Othejfs J.acked statiistxr 
cal and experimental k#w-how, and some 
mentioned lack of mana|fement interest, 
in addition, respondenis stressed l^ck 
of management support ^^nA difficulties 
with selection for tr|ining. (See p. 25.) 

GAO v^eoommends that Commiesion veem- 

phasize that it is departments' and 

agencies' primary riBVonsibil%ty to 
control the tvaininB of their employ, 
ees and to evaluate} the effective- _ 
• ness of that traiAg • GAO also . s 
recommends that thd Commission prpmxiie 
successful evaluation methods among 
the agencies andMfthrough its per- 
sonnel managefnenitj evaluation func- 
tion, monitor thJ^: agencies' evalua- 
tions of trainiri^. 

The- commission ^pund the .GAO survey data 
disheartening bfcxt said that, unfortunately 
it confirmed 'tl^e Commission's suspicions. 
The commission- added,, "Hopefully, the 
report will le^d to the formulation and 
impleraentatioii :of specific action recom- 
mendations." . 

■• iii - . 



To fullfill iU role in pwviding tra^n^ng 
leadership and guidance, GAO recommenda 
thai the Commission work with. Federal . 
de'p<xrtments and agencies on; the findings 
and. problems identified in this report 
and determine steps "to he taken for im- 
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-The basic statute authorizing employee training 
throughout most of the Federal Goverx>inent is ..title 5, 
united States Code, chapte^r 41 (5 U/S..C, 410l |t seq. 
(1970)). Executive Order 11348 of/April 20, 1967, gxves 
agency heads additional direction on how to use the general 
statutory authority. Both the law and .the Executive order . 
authorize the Civil Service Commission (CSC) to xpsue 
regulations governing various aspects of jtii^. law. ^ 

According to the Government Employees Training Act 
(GETA),.. public Law 85-507, July 7, 1958 (now 5 U.S.C. 4101 
et sea. (1970)), each department and agency head has the 
'^il^Y responsibility. for conducting training wxthxn that 
depari:ment or agency, 'n.e above mentioned ^xecutxve^order 
requires each ,agency head to plan, program, budget, operate, 
and evaluate .training program^. The specif i9 responsxbxlxties 
plaged on department and agency heads by -the law and regu^^^^^^^.^^ 
lations include: • 

_ ' •' ' 'J ■^-'^ 

—Determining the department's training needs^ 

" —Establishing aAd operating training programs to ^ 
meet those needs; 

-Establishing the criteria for. the selection of em- 
pibyges, for training. " • ' .' . . 



J-Determining the method and dicterit.- to whi.ch the 
departnieftt will finance- training. • *. 

. —Evaluating the resui^s„o^f trainxng. 

—Reporting to CSC "on training activities. 

under GE^A and the Executive order ,. CSC . is responsible 
for- providing, leadership "and guidance to Federal trainxny 
Wivities.' rfhrough its Bureau of Training, . CSC: 

' ' ■ ' , . 0' I 

■" —Plans and promotes the development, improvement, \ 
" . 'coordination, and. evaluation of training activitxes 
, under the law. ' - , / , 

■ . ••■ , ■ 10- ■ ■ //• 
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-Assists agenfcies in the development pf sound programs 
and financial plans for training, with par ticular^^-^. 
attention to planning; programing, budgeting, operate 
ing, evaluating, and iipproving training programs. 




— Provides for identifying and disseminatii 
of research in training ^echnoloc 

— Issues Government-wide trclining^^gulations • 



SIZE AND -COST OF FEDERAL / 
CIVILIAN TRA^INING PROGRAMS 



?RA^N] 



m fiscal year 1973, abo^t 960,000 civilian employees 
received a total of almost 4o million hours of training 
costing approximately $216 /million.! (This does not include 
trainees' salaries, which/are not%reported/tb CSC, If 
estimated salaries were /xnclUded, total training cost wpuld . 
be approximately $500 ijja^llion.J. Agencies report employee 



training under four general 



/ 



— Medical/, scientific, legal, 
engineering and related fi^l 
(professional). 




ones: 



f 



— Trades and crafts, facilities aildv 
services, and "how to" courses in 
administrative techniques and skills 
(technical) • • ^ 

Administi^^^lon, management 
supervijrion (administr 

Clerical and office se 
orientation , safety 
land communications 

' Total 



Approximate 

number ' 
r tra ihed 



150,000 



3^1,000 



211,000 



■^Bascal year^ figures are tl) 
yat the tirae of our writing 



imQ 




ilalDle from CSC 








./ Most trainees (71 percent) attended courses provided 
•j4 their own agencies. The remainder attendeja ihteragency 
or non-Government courses. y ^ 

The total (approximate) cost of this/train'ing in 
fiscal year 1973 w^s:- 



Tuition and fees 
Travel- and . per diem 

:Bo^ks/ materials, and. other related costs' 

Sal/ries of persons engaged in employee 
iievelopment 2^d training 

rimated salaries of training participants 
Total / . ;, " 



$. 32,000,000 
52,000,000 



.7,000,000 




115,000,000 

/' / 
95,000.000 

$511. 000', 000 



REPORT OF SUBCOMMITTEE ON MANPOWER 
AND CIVIL SERVICE OF HOUSE POST - • 1^ 
OFFICE AND CIVIL SERVICE COMMITTEE 

In 1967 the Subcommittee studied tl^, effectiveness of 

the imp]/Cinentation of 'GETAJ To determine the ex-tehtT to whicl 

training was evaluated, the st^ai^^visited field offices an§ 

met'witii training representatives at various department and 

agency headquarters. ^ - 

^In their report, they suggested that: ' . I - . / 

- y 

"* * * evaluation of training sjlSJild lG^i^ail±y^beg±nr: - 
with the determj-nation of a^i^egltima^^lieed for train- 
ing in^'the first instance. * * * thei): alternative 
in^hods, and their estimated^ alternat4^7e costs should 
'^e evaluated. - * * * Next cons iderai^on"- should be 
given to ev^u^ion of ^e traii^ing/ itself. * * * The 
evaluatj^otritep following this wbuld be a determination 

„eempl'oyee ' s performance on pie job after train- 
t:Sg as related to his ' performance prior to the train- 
ing." ' . 4 ' ' ■/ . 



* ■ fhe repoc-t concluded that most departments and agencies 
aoparently had no adequate training- evaluation process 
which emotively encompaased all the evaluation areas sug- 
gested" -.above . 

< • " ' / ' - , 

Additionally the' repojrt said:. . ' - 

, 4 

"Evaluation is 'the one ^rea of training programs that 
has probably suffered mOre than any other. Training 
needs have not always b^^n properly evaltlated. Methods . 
' of accomplishing . training properly have not always, 
'been properly evaluated in light of the training need.^ 
■ Training courses themselves have not always been prop- 
erly evaluated as to the specif ic .objectives desired 
by the manager sending employees to these fcourses. 
Employees' performance on the job is not bkj^g properly 
evaluated in terms of the train'ing which the employee 
has received, presximably to impro^ this performance."' 

_Tlie Subcommittee agreed that evaluating training is ^ 
a complex problem^ "However/' it added: * >^ 

"The fact that it is complex is no excuse for n^t 
making a reasonable effort to evaluate^whether or 
not the- Government is. getting a dollai*s worth of 
end product for each doll^ it spends on training 
its employees ^ ^ 

The report recommended that: 

— ^Departments and agencies give concerted attentio.n 
to developing training: evaluation .programs which 

* cover all phases of training from training require- ^ 
ments to empj-oyee perforraanqe aft^r training* 

— CSC take leadership in developing more definitiv.e 
guides for training evaluation and^ h^lp the depart- 
ments and agencies implement these guides • 
* 

1971 RE PORT ON TRAINING . ' 

•* 5 * ' 

on May'-25, i971, we issued a report (B-70896) to the . 
'CongTes^^^rT^mprbVfements needed in the management of training 



lander^ €ETA ia" tbe Departnent-of nefense . (DOp) , _ .In response 
td this report, which identified the same weaknesses xn 
accounting for training costs and making evaluations as the 
1967 Subcommittee report', CSC said: ? ^ . 

"AS outlined i^ previous- Gorrespondence with GAO, the^ 
commission has begun to fulfill its role in the planning 
and management of training areas and is -acquxrajig the 
knowledge and technical skills necessary^to serve in a 
" consultative and advisory capacity to FeHeralr^a^ncxes. 
in addition, ■ the Bureau of Personnel ManagemeW^alu- 
ation, in its review of agency personnel func^^^, 
'now examines such areas as the responsiveness/^of traxtt- 
iixg'to mission ne.eds, the responsiWlity-^or ^raxnxng 
. to mission needs,, the responsibil^/^^r traif^ng need 
< determination, development of schedules and prxorxties, 
equal training opportunity foi^^klnority ^oups and 
women, counseling for selfAde^glopment. anS^dv^ncement, 
lanagement attit.udes toward turning, and tYaihing 



m 

evalUatidh'' . ; 



METH0D6L0GY F&R 'ojLQ REVIEW ' " ^ ^ 

TO examine the progress made in>[^lementing-th^ recom- _ 
i^ndationS of -hoth'the 1967 Subcommittee report and our 
1971 report and 'the current state of training evaluatxon, 
.we^ent a questionnaire ;.(sfee app. I) to randomly ^^^^<^f^ 
^ training and employee develojJment officers .throughout the 
FedeTSMxecutive departments.?-. These departments accounted 
for about 82. percent of the Federal employees who recexved 



-tiraining^ in. fiscal year 1973. 



^-Executive departmenj:s , " as used xti this report and as. ^ 
defined in the,- United States Sovernment Orgartizatxon Manual 
1973-74, include, the Departments of Agriculture? Conmerce; 
Defense' (includingl the Air Force, Array, Navy, and o-dier ^ 

Defense agencies VTV^^^^-^^^S^StiS^^.,^^^^^^ 
.and urban Develobm^Xt; tl^ Interior; Justice; ^fbor ; State. 
. Transportatidn; and 'the-'Treasury. . 




Our sampling^^^^^focedures were planned so tbat reporting 
could be analyzed by- department and overall. In preliminary 
analyses we found few significant departmental differences, 
arid are therefore reporting our overall findings. 2 



Ovefall/ the responses contained information on approx- 
ircately 300 courses. 3 v7e asked respondents to answer the 
qjaestiohnaire for the course for which they are responsible 
(excluding correspondence courses, CSC training, and long- 
-term non-^Goyernment courses)> which received the most' effed- 
t iv3 evaluation • Because the results represent what the 
respondents believe to be their bast evaluated courses, 
their other ^courses must receive the same l«vel of evalua- 
tion, less thorough .evaluation, or no evaluation. 

The questionnaire was constructed after a review of 
Federal training rgcraiations and requirements, a review of 
training literature, a\jd discussions with training author- 
ities. It was designed to learn the'sBxtent to w|>i?ih train- 
ing and employee development oificewrs, (1) identify training 
course costs ^ (2) plan, 'review, anq Vevise 'training cptirses] 
(3) assess and evaluate the e£fectiveness 'and- benefits 
training, and <4) have projDlems wiiih training and its eva3 
ation. - . 

We discussed the survey r.esults with Bxxre^u of Training 
officials and have, incorporated their comments where appli- ^ 
cable,. • . J ^ , ' 

,Our conclusions and recommendations are in chapter 6i 



O 



^In this .report, "significant difference" refers to statis- 
tically significant differences determined by use of the Z 
test at the 95-pe?cent confidence level • See appendix III 
for a description of this tci.^e, ^ . " 

'^See appendix II for sampling, procedures. 
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fif rnPE OF REVIEW . ■ 

— - I ■ « * 

» 

The questionnaire was sent to. 881 randomly -selected 
training and employee development officers from all Federal 
Executive departments. Discussions were also held with 
CSC officials and with authorities in training evaluation. 




. CHAPTER 2 . 

• . , . * , COSTS -^OF TRAIHIHG .... ^ 

The 1967 House Subcbnunittee reported that most Federal 
departments and agencies apparently did not have adequate 
systems fot determining and reporting accurate training coat^. 
The report recommended that: 

"Departments and agencies should donsider establishing 
•better systems for keeping cost records of training 
programs. This should probably be incorporated into 
.existing cost accounting systems. The Civil Service 
Commission should coordinate the program to assure 
^uniformity and comparability." 

CSC ,said, in response to our 1971 report, that: 

"The major deficiencies cited in this report deal • 
. primarily with the absence of adequate financial 
management systems for. training in .the DOD activities 
audited. ' Unfortunately this problem is' not unique 
. to DOD — most Federal agencies have this same problem . 
We feel that the ireasons why' this -is so are wof^th 
some discussion. Large scale training of Federal 
employees is still relatively new in the Govern- 
'ihent and expenditures fbr such training hav^ grown i 
over the past decade. Training management systems 

have not kept pace Vith this growth for a host of- - - 

reasons, e.g;^; higher agency priorities, lack of ^ ^ 
systematic analysis, manpower and budget limitations;, 
and lack of top management concern." (Underscoring 
supplied. ) * . " , " 

CSC also said that it did not believe it would be 
prac£ical for DOD or any large Federal organization to' ^ 
require that training cost items be identified in accounting 
systems l^t that it would be possible for agencies to de- 
velop analytically derived and periodically adjusted cost 
estimates which would' be adequate fbr training ^irtanai^ement 
purposes. * * - - 

• i \ / 

The Federal Personnel Manual' (FPM.) -says that/ where 
'feasible, agencies should use available analytical tools 
to compare the *costa of various training solutions' and the 
values to be derived from them. 

The use qf cost data for evaluating training is also . 
addressed in the Bureau of Training's May 1971 pamphlet 
entitled "Training Evaluation: A Guide/ to its Planning, 
Development and Use in Agency Training/ Courses, " This 



pamphlet suggests that, in general, the more costly the 
cZlsi, the more comprehensive its evaluation ought to be. , 

Recognizing th^ necessity -of gathering cost? as an 
■important first tool in assessing training, a^n fiscal ^ejr 
1972 CSG developed the Training Cost Model • (TCM) which is. 

■»"* * * a simulation model for agency training management ' 
use in- making specific cost forecasts .with respect to 
training operations. Its. potential applications range 
from permitting agencies to make accurate performance- 
linked budget inputs for the training ■ function to . 
developing estimates of the cost of a proposed training 
course . " 

'esc believes the most significant use of all may be: 

„* * *foj, providing a sound basi^ for including training 
as a positive element in an agency's strategic^ lanning. 
That is, as ^agencies look forward in time anticipating 
the veijy substantial changes that are inevitable-, it 
Permits them to cost. out training as one significant 
potential change factor; to cost out the implications 
of any strategic decisions that involve training or.. 

trainincr-f or -changer and to ^^iclude 4:hat ^thinking zn 

a ve?J positive and concrete way in planning for ager^cy 
resource, allocation over the coming years. ; 

Despite the complex concepts underlying TCM, CSC claims 
that its-actual operation is simple- and that, once the re- 
quired basic assumptions are laid out, the calculations' can 
be idade- by clerical staff or. can be coftiputer programed. 

■ TCM calls for identifying each of the following impor- 
tant cost elements: . 

r-Total direct labor costs for staff associated with 
training. ' . . , 

r — Tuition. • , .• ' 

— ;Trainee salary costs. 

. — Traivel and pec diem.. . . '. 

—Other expenditures (books, materials, contractors,^ 
rentals, and related cost^). . 

We examined the extent to t/hich th.e Executive departments 
> accounted for the above cost elements and used TCM. 



ACCOUNTING FOR TRAINING COURSE COSTS 

Although some respondents made no accounting of training 
costs because "the cost of training would appear prohibitive^'' 
or "management does not seem to care/" 77 percent d.id some/' 
accounting- Of these, some explained that they would h; 
no reason to collect travel or per diem costs if the^^<5ourse; 
..on. which' they were reporting, v/as taught 'af-<the-_weric^^ it e or 
,they would not always collect 'tui^on and fees from their 
bureaus or divisions for ^courses- which they sponsored in- 
ternally but, rather, would includ.e such costs in their 
training office overall budget- 

Specific costs were accounted^ f<5r" as follows. 

^ Percent of respondents 

(note a) 



67' 
.60 
61 

< 

78 

82 
81 




Direct labor J2(Ssts for staff 
associaj;ed with training' 

Trainee salary costs (note b) 

Tuition and fees 

Travel and per 'diem 

Other expenditures (books, materials, 
contractors, rentals, etc.) r 

Total costs 



^"Respondents" refers to the 77 percent who accounted for 
training course costs. 

^Constituted about 58 percent of the total amount spent by 
the Federal Goveriment on ^training in fiscal year 1973. 

♦ Other differences surfaced as to what constitutes total 
cost. Eighty-one percent of the respondents indicated that 
they accounted ^Cor' total costs. However, 22 percent of these 
3;espondents did^^ hot account for, the salaries of their training 
staff* Instead, this cost was often iricl.uded in tuition 
fees of non-Government short-term and interagency^ courses. 

Sixty-six percent of the respondents who, accounted for 
total costs included trainee salaries" and 59 percent accounted 
for both direct labor and trainee salaries, which together 
^made up 80 percent of the cost of Federal employee training 
in fiscal year 1973. 
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USE OF TCM ' • 

who did not, iv perceni: -^i^ it was too coirt- 

ew co^SS^" and "no costs other than salaries. 



some respondents had ^°ft^^,J^-J,^^L?r??aining courses 
TCM. They^.thought it was not suited to ^heir t ^ 
because the^ courses' subDect matter ^^^^ot ..^^t 
measurable. Some said. The J^^J^^^^^/^^ perc;nt of those 

tSSlrt°„/?i^!Sr/sfirrJs,"r'e;iere?^eA 7S parent of 
those who used TCM.- 



^ CHAPTER 3 



PLANNING. REVIEV7. AND REVIgtON 



PLANNING 



FPM requires each agency to review^geriodically, but no 
less often than annually, its program to iSe^itify training 
needed to bring about more effective performancealr^bhe least 
possible cost. The program should contain cdmprehe'nsiye 
course plans which training officers are to follow* / 

The Subcommittee report said that: 



* * consideration should given to evaluation 
of the training itself, including such factors as 
the ^length of the training course, ^the adequacy of 
instruction, the course content and the relation- 
ship of this content to the predetermined objectives 
of the training; adequacy of training facilities; 
and adequacy of training ai-ds, etc.*" 

CSC materials suggest that a training -course plan include 
written course bbjectivets^ the content of the course, the 
method of presentation, how learning is to be measured, and 
procedures for cqVirse modification. Training authorities^ 
also suggest that a plan specify qualifications for trainees 
since the principles and content of a course may be excellent 
but not necessarily useful to some employees. 

Our questionnaire examined nol; only overall but also 
specific components of course plafis, ju^t as we examined both 
oveifall and detailed items in cost collection. 

Authorities agree that a good course plan should address: 



Organizational needs : The knowledge^ skill, and .aBllity 
requirements for maximum effectiveness of the agency opera'tions 
which the course addresses. . 

Training objectives : The goals to be reached as a result 'of 
the course. The CSC Guide states: "Ideally, they should be 
stated in '*perJ§t1b:mance terms, i.e., some actions or behaviors 
which the trainee is expected to -exhibit as aT result of his 
training experience." ^ • * . 



C urriculum or content ; The specific topics or subject 

matter to be covered. r i ' 

ti % 

pro graming of the course, i< or lesson plan, and the cQuxse ' 
material an d equipme"Htl The methodology" and timing od:. .the 
presentation o£ the subject matter (e.g., 3 hours' lecture, 
1 hour of slides, 1 hour bf discussion, and such necessary 
equipment as a slide projector,' flip charts, etc.). 

Tr ainee qualifications ; The prerequisities that a trainee 
must have (tninimum reading -rates, mathematical skill, or 
machine familiarity, for example)., designed to restrict the 
course to trainees who might benefit from it. 

critiques r elating to spedxfic parts of the course ; Trainees' 
reactions to specific features of the course in tne form of 
comments that can be tabulated and quantified as an early 
step in the course evaluation. ' . 

Me asurement of learning ;- A description of how the trainee's 
learning will be measured 'both during the course and upop 
completion. The instruments .developed for these measure- 
ments and specified in/the plan are usually in the form of^ 
-written performance tests or informal or formal demonstrations 
or' discussions. • ^ 

Validation and evaluation ; A methodology for assessing the 
, adequacy and suitability of the coursei ^ ■ * - 

• Updating procedures; How to make changes on the basis of 
observation or performance measures'' and how to reexamine ■ 
the original objectives to improve the course. ' 



* Ninety percent/of the re::sp6ndents had written training 
course plans;- however^ the individual components of a good 
plan were not aly/ays included/ as the following table shows. 



Percent of respondents 
( note a ) 



Organi^tion^J. needs 

Training objectives 

Curriculum pr cont/nt 

Programing of co)o:s( 

Course materia] 

Trainae^'^ali^cati^. _ _ 

Critiques relating to parts of 
course 

Measurement -of learning 
during course 

Measurement of learning u^on 
course completion 

Validation and evaluation 

Updating prbcedur^ ' 



65 
91 

85 
72 
75 
- 47 
62 

51- 

51 

48 
56 



0 



^ '^"Respondents" refers to the '90 percent who had 

written course plans .^^.^ ' . ' f 

"Host plans covered the conducting o^:^ tfiie course (i.e., 
the objectives, curricului^^ materials an^d equipment, and 
lesson plans), but inst;ri^1.<tins for measuring course results 
were less often included* / * 

, ... 

Less than half the respondents had trainee qualifications 
ift their plans/. Authorities suggest that such qualifications 
prevent the course from being used as a "reward" or a , ' 
temporary "dumping ground," which some :fes pendents mentioned 
as reasons for selecting emJ>loyees for 'tjraining. 



Over 60 percent of the' respondents planned for "cwrse" 
critiques. Although some studies show that' trainees who 
enjoy a course are likely to benefit from it, critiques 
do not necessarily measure any learning that has taken ^ 
place. Nevertheless, training officials said that critiques 
are often used because t^ey can be easily administered. 

Less than half th6 respondents planned for course 
validation and evaluation.' While authorities acknowledge 
that the scope of this process may vary with' the type of 
course and rejggjces available, most agree that it should 
be in the overall course plan to the extent practicable.'. 

Strengths 'and weaknesses of a course are discovered 
when it is evaluated after cojgpletion. Information fathered 
and a reexamination of the original objectives are used for 
course modification. ' Training officials deem ^t essential 
to the success of future courses to plan ahead for feedback 
and necessary changes. Only 56 percent .<^the' respondents/^'" 
however #^ planned for updating. ^ ■ y 

REVIEW AND REVISION ' • ' 

Authorities suggest that measurements, of the effective- 
ness of training should-be made at three times: duri?^ the 
training, upon completion of the training, and after return 
to the job. Sixteen percent of the respondents took no 
measurements, 22' percent measured perfoinrtance at one of thes 
times, 24 percent at two of these times ^ and 38 percent at 
all three times. . ^ ? ' 

' . - , ^ \ 
Performance measures during and ' , \ 

upon completion of course • . ^ 

The CSC Guide stresses incourse evaluation \o insure 
th.at course objectives are being' met. Along with current 
literature oi> evaluating training, *it ders6ribe.s, as' shown 
in the following table, bo^th direct and indirect methods to . 
test performance during' arid at the end of' a Course.* 





Indirect 

' , 

opiriion questionnaires 
lassroom observa-t^ion 

trainee conversation's 
instructor "feelings " . 



There a^e limit irig features to these methods; for . 
example^ (/) test construction is dif f icult, ^ (2) partici- 
pation in/class discussions may not demonstrate, actual 
learning; (3) opinion questionnaires m^y reflect personal 
biases/ and (4) observers must be familiar with the subject 
ma.t%ej 

The timing of the perf6rmance measures may depend upon 
th^ type of test used and such course features as lengthy 
s^ze^ material tavight^ and riuWber of times the course has 
ifeen given> For e^cample^ if an observer is used^- his 
y^orran^xits may be given at the end t)f the day; 



a critique 

may be used only^ at the end of 'the - course; in a lon^ course^ 
pencil and paper' tests may b^ given at tlie end of eab^ unit; 
and task performance tests may^ be given dail^ if the^ next 
step depends upon the mastery of the previous one. . 
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Fifty-eight percent of the respondents gave periodic 
performance te^ts during the course and 55 per.cent di^ so 
upon completion/ as shown belqw. 

^ ' ""^^ ' --^ 

Percent of r^gsporfdents (note a) 

Pencil and InstructQr'>*Msk ^ / 

paper test ratings test Other ; 

' ' ' b 

^ TJuting-thB^eaurs^ 65 ^ ^ 46^ 63 . 10 , 

* ' b 

Upon completion of 61 , ,46 46 T • 

♦ course ^ 

^"Respondents" refers to the 58 percent who. took measures 

dSring the course and the 55 percent who took 
. measures upon completion. . ^ 

^Of those who answered "other," most used oral 'exami- 
/ nations, role playing ^ or peer^ critique^- ^measure- 



ments . y 



Ori-the-job performahce measures" ^ Jk** 

^ ' , " • ' ^ - ' 

Tests during and at the completion of training courses o. 
measure learning from the course ; feedback from these tests 
helps improve course strategy ;or methods. Evaluation uppn 
return to the job tells whe^ther and how learning is applied. 

Some trainers' consider, assessing the trained^'s job 
performance the most criacial phase of eva;Luation because 
it shows both the adequacy ai^d the relevancy of the training; 
a trainee may master techniques and principles but be unable 
to. use them- in his daily work. Thus, an important part of 
the evaluation is determining the atnount of transfer from, 
the course to the on-the-job situation and how long the \ 
resulting changes last. Also important is the examination": 
of organizational changes resulting in improved supervision, 
product quality, actual sa^J-i-ngs in the department 'orj^" 
improved morale and job satisfaction.. Seventy per^t.of". 
the respondents said that performance was assessed ,on the 
job after training. , > * 

Of those making this assessment, about 70 percent ,used 
supervisors ' ratings and job performance measu'tes . Forty 

• 26 ■ ■ • 



t)ercent used graduate ^ critiques cunid 13 percent said they 
used "other" techniques— prijoarily informal observati,ons, 
discussions, and interviews with training staff . 

While immediate checking upon retxirn to ^ job may 
.indicate some learning, training experts consider a latex 
evaluation which indicates retention of learning and vise in 
the everyday situation a more valid, check* Many atithorities 
say- that the posttxaining appraisal should' be made after 
^3 months or more so that the trainees have an opportunity^ 
to practice what they have learned. Additional measixres 
taken later can validate the findings of this appraisal. 

* • ... - 

* Tiie respondents measured on-the-job performance at 
jiifferent periods of time after training, as follows: 

* — - 

Percent of respondents 
( note a ) 



Less than 3 months 



55 



3 tc^ 6 months 



^7 



46 



— r 
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.6 to l2 months 

■ ■ ■ L 

After 12' mot^ths 

^"Respondents" refers to the 70 percejjt who assessed on- 
the-job performance after . training • ' ^ ' 

Trainers also consider accurate results important in^ 
measviring the effects of training. This means ^at measures 
3hQuld not only be taken repeatedly but also tSffat bfvrthe-job 
performance should be measxxred in detail. 'Fifty-five 
percent of- the^ respondents rated the trainees ' perfoi 
satisfactory or aot satisfactory, and .42 perc^t Jtated 
performance on a graduated scale (e.g.*, very goodT^ good, 
fair, poorr, very pool?, or from 0 to-lOO percent). Eighty 
percent rated the trainees'' job performance in relation to, 
specif io/parts .of tlieir jobSc*"'" 



L ' I ' - ' ^ 

VPeirceht" refers to the 70' percent who assessed^ on-the- 
job penformahce after training.. • 



Trainers consider evaluation a starting point for 
co-^rse improveiue^nt* '..p;eed}:)ack can Xead to correcting weak- 
^ne'^ses. Ninety-^three percent .of the respondents formally 
modified, or updated, their training courses for the 
following reasons • . " * 

^' *^ 

\ ' Perc&t of .'respond6nts 
( note a ) 



Change in current trends ^ 

— Change-in preestablished , policies / 
or procedures 

Change in goals, operations /*and/oje 56 
' conditions * ^ 

Result of specific problems 56. 

Result of benefits 34 

Hesxilt of j^bb^ perfoinnance-ratdngfi^ — ~2r2r- 



Change in/ or appraisal bf, ^ 21 

-performance 



Result of course pretesting . . r 12- 

All others (primarily . Student .and/or 12 „ ' . ) 
- instructor critigug^^ and.fivaluatipna) - ' V 

"Respondents" refers to tb.e 93 percent who iiKbdified . ^ 

«their coursed* . - ' 

• * : ' , * ' " . ^* \ . . • V • 

- The CSC Guide suggests tji'at data 6k trainee achieve- 
ment be ^collected continuously;and systematically and _ 
followed by appropriate course, modification.. 

t^aa than, a quarter of the "^ahal^ea training cotirses, 
however, w6re Wodif i^d' as a result' of a trainee's .suBseguent 
onr-th^-job. i)erformancf This couia be expected when, as 
-previou&ly noted^ ,6n3Ly lialf. the trgiining ,'qours^ plans . . i J:_ 



providea updating prop©4ures. and less than 60 ■percent of .the 
respondents measured Effectiveness during or upon completxon 
of the coutse^ ' , 

Respondents said ths^t some courses were changed to .^- 
'keep the trainees interested.. If a-coxirse was P9pular, its 
existence' was guaranteed whether or not the trainees 
actually needed it. This also kept the training budget 
and staff growing. . ; 



4r . • 



29 



20 




. ' ' . . • CHAPTER 4 ^ 

. . • . ^ EVaLUaTlON TECHNIQUES 

STATISTICAL AND EXPERIMENTAL APPROACHES - 

Reliable and rigorous evaluative research designs 'which 
* can be applied' to training have been (aevelo|)ed* The ideal 
methodology requires meastiring change from before training 
to after? identifying to J:he extent^.possible cause and ef-/ 
fect;^ and using statistically equivalent/ randomly selected 
experimental and control groups* When random selection is 
not possible, equivalent results can be obtained by giving 
precourse testing to. two groups to determine the differences 
between them and then training one group and using the other 
as a control group* If u^ing a control ^roup is not possible 
a series »of precourse .and postqourse tests m^y'be adminis- 
tered and compared to determine the . influence the training ^ 
hf^d ori performance. . * ^^i " - ^ 

^ Nineteen percent of the respondents usee? statistical 
or experimental methods to evaluate the effectiveness of 
their trainihg courses.. jOf .these,, oyer half (55 percent), 
used the pretest and posttest method without a .control - 
group/ Training officiaTs and evaluation" e^^^^^ 
this method is weaker than method s using control groups. Th e 
following table shows the various tec^iqu^s used by the 
19 percen€>\.- • 



Percent of resipondents 
( note a ) 



8 



Compare the job performance of a large 
. (100-500), random, nonbiaaed aample tff 
graduates with a similar sample of 
workers who have, not received training . 
but who have b6en on the job as long 
as the graduates . » ' - 

Compare the on-the-job performance of a 

tZiKt^^^^ °^ graduates ^ (less than 50) 
With the on-the-job performance of a 
axrailar sample of workers who have not 
had recent training but who have be§n 
on the job as long aa the graduates . 12 

Compare the on-the-job performance* of a 
repreaentative group of graduates with 
the performance? of a matched, or non- 
biaaed, Randomly aelected control group 

' ^r";^*^' -^^^n working on 

the D<Sb,the aame length of time. (a 

s?™?r^ r^""? ^ °f employeea 

Ittl^l • ° with re- . . 

spect ty, relevant peraonal characteria- 
tica or attributea that affect job per- 
4°a^;::,^"^^ ^° receive ' 

6 

Adrainia'ter a preteat and.a poatteat to ' 
compare the job performance of a group 
of people befbre training with their job 
performancea after training (no control 
groups are uaed>=. . - "^"oj. 

Administer a pretest and a poatteat to 
both a training ^roup and a control group 
(Dob performance acorea are taken on both 
groupa before training and again after 
training). / ^ 

Other~2tn analysis of ^the "other" tech- ** ' ' / 

niques indicates that these re- 
spondents considered post course 
questionnaires, supervisors' re- 
ports -on trainees, interviews 
with graduates; and similar inea- 
^ • ' surements zlb statistical or experi- ' 
- -mental techniques. These, procedures' ' 

are not considered such by most ' * ^ 

training experts . . . 

■ ' V 31 
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ANALYSIS Of BENEFITS 



FPM suggests that, wherever feasible, an agency make 
cost-benefxt analyses to determine how av.ailable resources 
can best- serve the agency. Many factors influence how indep^h 
^5^,^"^- Y!^^ should be. Some course criteria mentioned in 
CSC s Guxde are - (1) the content (can the benefits be measured 
wxth any precision?) , (2) the. learning level (skill training 
may require a more thorough evaluation- "than kn orientation 
course"), and (3) the costs. The Guide suggests that the more 
costly the course, the more compreheng-ive the evaluation ought 
to be. Two other factors ^discussedj/n the Guide are: 

(1) The degree of control exercised .by the agency over 
course delivery: where this control covers all elements- 
medxa, facxlxties and, in particular, the instruction 
staff — the level of evaluation may be set based on 
consideration of the above variables. .However, where, 
it is desirable to„use frequent guest faculty to present 
the program, control over the delivery stage will be 
limited. As -a consequence,, it will be more difficult 
to apply achievement measures. 



(2) The resources available to the agency: a full scale • 
.evaluatxon plan represents- .considerable time, money 
and professional competency on the part of the ' 
traijning personnel involved. Since increased 
reliability comqs at "a high price, it must be 'decided 
^ whaij poxnt on the , continuum constitutes an acceptable ^ 
trade off between cost and reliability. This decision 
makxng xs further complicated because it does not 
Aormally involve only a single course, but rather 
every internally developed course, making up the 
agency's training program. 

To help agencies evaluate training, the CSC Bureau of 
Traxnxng developed Training Value xModel I (TVM I) , the first 
^^^^Planned series of models. Aqcording to the Bureau, 

"* * * relates specifically to training designed to 
, improve perfortnande of those work activities that result 
m measurable proaucts or service?, initial estimates 
suggest that the model has potential "for application 
• to the daily activities of nearly one million Federal 
• emp'loyees. Utilization this process will enable 
management to. assess whether training would significantly 
xmptotre employee efficienOy.. Faced with a choice of 



several training 'alternaibives, the model will also 
provide the manager' with necessary information for 
selecting that training which is potentially most 
e^fedtive.-" 

Because TVM I was developed 'only recently, our question 
naite d5d ndt ask if it was used-. We did ask if the training 
Eei^its we?e analyzed and to what extent- Forty-seven per- 
cenfanalyzed benefits, using the procedures which.f ollow. 



Percent of respondents 
(QQte__a) 



Identify benefits- • ^"^ 

44 

Measure results 

Measure benefits ^ • 

Measure dollar value ' "'"^ 

Cosfffe)enef it. ratio • ^ 

a-Respondents" refers^ to the 47 percent who 
analyzed benefits. 



CHAPTER 5 



PROBLEMS WITH TRAINING 



GENERAL PROBLEMS ^ • 

Almost 50 percent of the respondents had problems with, 
training. Over half said they had insufficient resources. 
Many commented that additional resources would allow more 
sophisticated C9urse evaluation which, in turn, would make 
courses more effective. Lack of resources was usually 
traced to lack of management interest. This appraisal is 
confirmed in the 1973 CSC report, "Disincentives to Effective 
Employee Training and Development," where the firs^t dijs- 
incentive cited is, "The benefits of training are not clear^ 
to top management." The report goes on to sta,te,thatr 

"This is especially a problem, because of t^ie lack of 
methods which currently exist to demonstrate potential 
benefits to managers. Without means to determine 
training and development benefits, top management is 
likely to concentrate its resources in areas where the 
returns are more evident." ^' 

Respondents cited. the following problems. 



Percent of 
respondents (not# a ) 



Insufficient ^ resources 



5-6' 



Results^ cannot be determined 



37 



Line management fails to allow wha^ 
' was taught to be used on the jdb^ 



31 



PrcT^uctivity cannot be related to 
training / 



25 



Training - reports too low in orgaxiiz^tiori 



14 




.lit 



• - learned 




Other (these re'spong.es dealt primarily 
With trainee selection and< a lack 
of support by managers and sniper- 
visors) . ' . 



22 



' ' ^"Respondents'* refers to thd 60 percent who had ^ ^ 
problettis with « training. , . 



Respondents' comments indicated that problems with 
traifiing programs were widespread among departments, regard- 
less of 'the size or scope of the training bfficers' res- 
ponsibilities. • The problems generally related to selection 
processes, management support', and evaluation techniques. 

Some respondents reported that training ^::ourses are. re- 
quired but may not be needed by the trainees. Many said that 
training was too often used as a reward or punishment for 
employees. One respondent said, "training is used as a ^ 
sop to young professionals who have been given unchallengihg, 
unrewarding jobs. Promise them a master's degree and tfiej"' 11 
stay.'' Typical comments were: "training ^for^ training' s 
sake with little or no regard . for' possible beneTits;" 
"people who actually need training may not get it;" "some 
managers see the^ need for training only when there is nothing 
else to do;": and' "a panacea for organizational ills." Other 
respondents stressed that emphasis was on education, not on- 
the-job performance. ' ^ 

Frequently cited was a conflict between a trainee's 
job goals ^nd those of the training course. When the trainee 
returns to ^the job, certain obstacles which discourage the 
conversion of learning to on-the-job performance may occur; 
supervisors and peers often resist new methods and dissuade 
the trainee from -using them. One respondent indicated that 
if ^the trainee's supervisor has not had the same training 
and does not want to change his ways, the^trainee is likely 
to be frustrated in applying his new knowledge and skills on 
the job. • Some respondents recommended that managers and 
supervisors play a larger role in planning and coordinating 
traxning with on-the-job goals, tjany respondents felt that - 
xf the managers' and peers' attitudes could not be changed, 
^training might be useless., 

EVALUATION PROBLEMS " • - '» * ^ . 

• * ■ * ,-. - ' • - ■ 

Thirty-h'ine percent of th^ respondents were dissatisfied 
with the level of evaluations. ■ Of these, 75 percent lacked 
resources and 22 percent lacked statistical', and experimental 
know-how. Thirty-nine percent gave additional reasons such 
as: "no real follow-up encouraged, "lack' of managenfent • 
support," dr. "apathy. from trainees and field line, super- 
visors." .Others reported that the course planning was t6o 
vague for evaluation.. (See ch. 3.) 

•Many resp.pndents said that few, i£ any, metihods'were 
available to evaluate the impact of ..nontechnical .courses 
because it is impossible to measure quality. Many urged .that 
■a S(tancl,ard evaluation procedure, be developed. 



' Many respondents wrote that they were interested in 
improving their .evaluations but were restricted to only the 
most rudimentary techniques by a lack of resources. Some 
assumed that if there was no request for detailed and exten- 
sive -evaluations, staff time and effort spent in making them 
Should be wasted. JSome questioned whjBther extensive evalua- 
tions would be used. .A final observation reflected the views 
,of many: VIn training, the concern is to-be able to show 
that something has been done, not that something effective 
has be'en done. " 




/ 



^* CHAPTER 6 

» / CQMCLUSIONS Al^TD RECOMMENDATIONS 

CONCIiU^IONS f " ■ 




^"^^guestionnaire results shpw" that^the^ :evalu^tion re- 
quirements of GETA, Executive Order ai'3t8> and the recommLri. 



rnno^?L^''^^^^^''^.^?^''^®"^° training cost dat^^ was /, 

Sn^o^S?f ^J^^^^wi^ly- -While- most respofidents collected, 
some 4ata, number were not doing so to the extent cnni?^o>.o^ 
necessary^ by training authorities for e^fec^iS^'^cost conJrS^ 
for example, some did not even identify the two larges? ' / 
elements of training cost~-trainee and trairting staff salaries, 

T^M ^i^?^^ ""^f ""^^^ CSC's TCM is surprising si^ce 

Sn^nvffS''^-^- ^^.CSC's fiscal year 1972' annual report and 
Employee Training m Federal Servige FY 1972." Its use iS ' 
also suggested in the FPM, and CSC has publicized' TCM^and 
conducts training in its use. , • . / * ' 

^'^J'l}-^°'^^^^^°^^'^^^^ respondents reported that they had ' ' 

training cours-e plans, the plans often lacked the 
• • ^ necessary for future course evaluation. 

Strengths in_ the. plans, wece in area^- of course -deliArerv-- 
objectives, curriculum,- materials, and lesson plans. Weak- 
^for^r''^-;-^''!^^ °^ greatest conseguence to course evalua- ■ 

J^^,";°^jf^?^t\°^rtrainfee quaXifiSations;- measurement , 
°^,-!-f^f ^^^5 during and upon completion of the course; atid 
validation, evaluation, and updating procedures. : 

Some performance measures were taken during the course 
some^upon its completion, and some after the trainee's re-' 

^rn to thep^b but the extent, the' detail, the timing, and 
^hf ff°Pirfu^^® measures, used varied widely. In many, cases 
the use^f these measures was; so" limited- that, effective^.'. ■ 
ev|lu^tion would h^ye b^en impossible. While training V 

-authorities ^ay it Inay be poss^le to' assess the effectiveness 

?f JhSrif^^S^^n"'^^' ^ traji^ritigue- or asking the t^KSI 
i£ they liked thecQjorse-^and only slightly- over half -thf^ ' — ^ 

Sels'S^rmS"' -aae,ua.te to detennine. hc„ th™se. 



Jj3 



37 
as 



Assessing the on-the-job performance of the trainee is . 
probably the most crucial step in evaluation because it 
demonstrates eoufse-«adequacx and relevancy. To make sat-, 
isfactory measurement of the bn-the- job-performance change 
possible-,, course objectives mus€ be .clearly planned ahd 
specified. Unless and until on-the-job, effectiveness • is 
measured against course objectives. and courses are , then ^modi- 
fied accordingly, the benefits of particular courses and, in 
turn, effective, management of training may be questioned. 

.Statistical and fexperiirfental evaluations are not widely 
conducted; those that are done are elementa^f. However, over 
60 percent o'f the respondents said that they were satisfied 
with the level of evaluation^ being 'performed. TVM T: could 
be helpful if TCM were used, more or if data on. the course, 
costs upon which TVM I depends' were collected more thoroughly. 

' Few; if ariyr Government ag^iicies have sufficient ^ . . 
resources to fund all needed, training. .Therefore, lustxfying 
the value of courses by citing on-the-job improvements is 
crucial to efficient . and effective Government manpower 
-management. » 

■ ■ . • 

RECOMMENDA l'JONS . 

, • ■ - The Chairman 'GSC, . Shbul(^.: -~ ~- -" ' - — 

*- --Work with Federal 'departments and .agencies on the ■ 
V .findings and proble^is identified in this report. 

^ -^-Follow up to determine steps ta.ken by the departments., 
and » agencies for improvement.- 

«-Reemphasize that it is the departments' ,,and a^nc'ieS'*' 
primaty responsibility to control and;evalAlate-"t:raining' 
jf' - for their em^loyees^ ' ;. ' - . . ; \ ~ • 

' lyjonitor the evaluation of training in .€hk departments 

and 'agencies and promote successful |(.e,thodB of evalUa- 
'•■ . tion..- - ' ' . y -' ' 

Specif. ic actions of CSC shoji Id Include: 

■ • —Defining, and achieving a^consensus among the Executive 
. ' departments and. age.ndies' "on elements' tha± make up the 
. ^ - total cost' of' training and then ' insuring that data on 
-these elements is uniformly d^tepttined and collecl^ed. ^ 



V ■ * 



— Determining why TCM has been little used, amending and 
refining it accordingly, and then increasing efforts to 
publicize TCM to training of f fcials at 'all levels. 

^ / 1^ • ^ ' " 

' —Helping the departments and/agencies develop written 
course plans which include /instructions for cpurse 
delivery, me,asuring resul^, and modifying and up- 
dating the i::ourse. 

MATTERS FOP( CONSIDERATION BY T^IE CONGRESS 

. This report should help t'^e Congress assess how wel 
CSC and the departments and agencies are fulfilling the 
evaluation requiremenlfs of GETA, Executive Order 11348/ and 
the recommendations of -the Subcommittee . report of 1967/. - 
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AOOMCSS 



CITY AWS> 5TATK 



THg MUM«R OF COURSES XH EACH OF T RE rOLLOf^.HG CATLOOmrS FOR .CHICH YOg A« BESPOHSI.LE OURlNG FISCAL Y6AH 



OHO. OF PROFESSIOMAL COURSES (MEDICAL 
SCICMTinC. LEGAL 4 RELATCO) 

OHO. OF AOMIMISTRATIVC. MAMACitHEMT * 
SUPERVISION COURSES 



i 1 "°' technical coursm 
P~[ mp. of ojher courses (clcrIcau, etc) 



DFSCRrpTlONXf COURSE SELECT£D1^R QU£STlONHAIR€ R6SP0HSE 



NAMe OF COURSC 



APPROXIt^TE COURSE OURATIOM. 



tOAYSJ 
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CATEGORY OF COURSE; (Chmck Off) 

Qpr6fE5SIOMAL.(MEOJCAU scientific, legal 4 RELATED) | TECHNICAL ^ 
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SPONSOR OF THE COURSE 



Qaoehcy Qimteragemcy • I |mom.govermmemV shorVtcrh 
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. NO. OF TECHNICAL COURSES y 

/ 
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- . --- 



PURPOSE " . .. 
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[^MECT ORGAN/ZATIONAL GOALS* INCREASE PROFICIENCY 

^ j~| IMPROVE Human relations * [ ^ \ safety 




PROVIDE JOB ENTRY SKILL, APTITUDE. OR. 
EDUCATION REQUIREMENTS 




rCfUONAL O^KLOPMWT 
I [ ORIgNJ/CriOK 



OTMtR tpmcltry ^ 



V^. FOR WHAT Occupational group is the course givcht (Chtck Onfj^^f) 



^Professional fAccouNT/^NT^. ENcmEtRS, J^''****-''"?'''^l*^°!li^iV.?J"/^^^ (nmrsesv 

DOCTORS. ETC.) ' * f l-J AIDS, i,A«bRAT0RY TCCM., RTC.) 



rr-^PRO. . 

I , } DOCTORS, ETC.) 
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CLERICAL AMqOFFICe WORKERS 
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% DO Vou «XKE THE roLLO«.HO TYi-E* or M*««.i.« eiimfxeii 



'U IS AH ACCOUHTIHO MADE OF TOE COURSE COSTSt • . . ' - 

."X-E THE .OU.O--C cpsxr oe'ter-heo rH. : .- , _ 

■ rivEsnHo -rE^&wL"Ah&g^%?'^?t^.^"H^^'lli*H?H^«^^^ DvEsg-- 

p L^^^L^^^^^^^^^^^^^^,^^^^^^^ Or«CDH0 T.AVELAHO.E.O.E- 




ARE CUEARLY S^EaritOT 



I ] ORCAMIZ^MAL RET£D5 



I jyiUlMlMC OWECnVES 
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TRAININO 
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T WE COURSEt 

\U IF YE5.-WKAT TYPE Of TE^TS ARK OlVEMl 

PAPER AHO ^EMCtL ACMFEVEMEUT TEST 

\ j iHSTWUCTOyS RATINGS' 



{□yes Qmp « 



• [~ljO« OR TASK PERFORMANCE TESTS 
r~ToTRER U—^t:^ 



.X .S A-1>E.rORN>^C E MEASURE O. TEST «VEH>T THE WMPLCIigH O. THE T,MHmO^»R.» > 

I 3 YES ' j ] ^«0 U«;f^t^ • » ^ ^ 

QpAPER^ENOLACNlEVEI-EHTTESt . Q JO, OR TASK l-t«rt>l«AHCE TEST, 



14. If YES. WHAT TYPE? 



a 



INSTRUCTOJI RATINGS 
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if.isTHeio0i 
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rnjO»PERl4yiANCE . ^ QsUPERVISO^'RiTINO 
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JAT<SrACTORY OR KOT «ATI$rACTO(^ 



a^i-r5L?i'^Kf i^f jenrowiAiice is rated oh a 
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• Ore, D. 
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eVAtOATlON TECHNIQUES 



^ Vl^S2£f.^" """"•"O*"'**- »«CT>«3(W Oteo TO EVALUATE 

TKt EFFECnveWEH Of THE TRAININO COOWEf "ALUATE 



rn^« r^jwo num.!, ,,34:^ 



n. IF YEJ. WHIM or THE rOU-OWIHO KnOCEOURES Ar£ tiSEOI f««* .J/ ^,,7 



•UT WMO NAve tecN 




P'* .^''^"^•'"'S.fgaZrc*^ W.T.aW:S=^^^ M.OM T«A.H.HO «Th' . 

□ * T*I5? S§rT^oroS,7.\i?S;?V-.2,'Sl}S, Vh'o- l^l^miSVS;^^^-* « ••TM THE TaA,HCE 



OTHER »^»(,; 



J«. ARE THE 8E«Er»TI THAT RESULT FROM THE COURSE AHALYZEOf 



Ves. IN WHAT OETAILI rOrtrt .« rt., ^/;/ " ^ '■ 

•a.pE«T..YT-HE,EHE«TS Q «EA«,RE THE «HEr.T,. O-EA^^RE TH. .E»UT. O, riit .EHE«T^ 
LJ«EA«,RC THE DOLLAR VALUE Or THE .EHEr,T, QhCAWRE THE CO»T .«.er.r »*r.O 



»^yOU 0.«AT«r,EO «TH the level or CVALUAT.OHf .EIHO PERroR«co,' 
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a. IF ycCrOU WHAT HCA$OM$» rOv#«k mtt M 

□ 00 MOY THIHK IT H'CCESSAWY TO tVA^'JATE 
TJIAJ.HIMC. f eVONp CWTIQU W „, ^ 



p**^ tACK Of STATISTICAL AMO CX^EHIMEHTAL KNOW HOW 
[~( LACK Of RESOUHCeS TO CA^WT OUT CVALUATtOHS * 



□ irOULO HATHEH mveST HtsOumCM »M AOWTIOMAL j— I qtHEH (Exfitiri 
TRAIMIHO COVWCS* ' , J 



2S« ir^YOUIIOflCANI 



ZATIOH I$ 0«VELO«NO A fYJTEM TO EVALUATE THAlMlMC. 1»L€AW OESCmiC 



Wy DOES THIS COURSE RECOVE VOUR >*OST EfFECTIVt CVALUATIOH? 
Q«ir COUWE QuAHAOtMEHT IHTEREST 



□ 



StCMiriCANT TO ACEUCT WSSlOlT" 



OTHER r^**M S^*<IM 



PROSLEJilSWtTH TRAIHINC 

JO, DO YOU HAVE PROSUtHS WITH TRAlMWOt 



NO utt^^mn* 



SU If YES, WHAT ARE YOUR^R0»LE«« fO»«* *«t 

I lUHEUAHAOlMlHTfAiUTO ALLOWWHJrrWASTAUCHT 

j I TRAIKINO RC^RTS TOO LOW W THE OROAMIZATIOM ,j yj^, 0>#-THK-JOg 

^. Q^Ee" OJWUW P^WURC COOHTtRS WAT WAS LKARNEO {"J R««LTS Of TfUI«mO;CAHHOT BE OTTt^WHlO 

QCHAMCES IH PRODUCTIVITY CAHIIOT VK RELATtO .( I msUf f lOEHT Rt|0«RC«S - 
TO TRAIMIMO ' - , j, V 1 

I | oTHERfyi»#»» tp—itri ' ' - ^ ' 

^ . Ill F I . * ' . ' ' . ' 

ADDITIOHAtCOMMEMTS 

ir v«M UAW AOWTIOM AL CO«HEHTS OH AMY Of THE ITCHS WlTHlM THt OOESTIORM AIRE OR TOPICSHOT COVIRIO. PLtAfl 
"^ixPR^sJ^^Si^EW «IOCOHHEHTS ARE ^JREATLY APPRCCIATIO. 
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SAMPLING' ME THODOLOGY ' . , ' 

SAMPLE SELECTION BY DEPARTMENT • ' 

Executive departia,ent training, officers gave us lists 
of their agency training offiqers, who in turn gave us lists, 
of their training and development officers. We sent question- 
naires to '881 of these officers, randomly selected, by depart- 
ment, from the total of 1,60,6, which was a ntmiber sufficient?:; 
to yield returns with an error rate of 10 percent or less at 
a 95-percent confidence level. 

SELECTION OF OVERALL SAMPLE 

To obtain, valid statistics for all departments combined, 
the responses were randomly thinned so that each department 
V7as represented in the overall sample in the s^e proportion' 
as its total number of training personnel to all training 
personnel in the executive. departments . (For example, if 
'Department A had 160 training personnel, representing la tier- 
cent of the total of 1,606, then 10 percent of the overall 
sample was randomly selected frdm all questionnaires returned 
frm Department A.) We included 3^90 questionnaires in the 
overall sample analysis. - ' 

' RETURN RATE • , 

Of the 881 questionnaires sent, 205 "were returned by 
personnel who were not legitimate .member's of the sarapl6 
population because they were not.^ responsible . for conducting, 
training or did riot conduct the type .of course about which we 
asked them to reply.. To be conservative, we took the worst- 
case position and assimied that the remaining 676 questionnaires 
were re"<ifeived by training officers who conducted tr.aining 
courses of interest to the survey and were therefore legitjjnate 
members of the sample population. Five hundred and seventy- 
one, or 84 percent, -Of these assimied-to-be-legitimate sample 
population members returned the questionnaire. The response 
rate would have been much higher, of. course, if we had not 
assumed that all the nonrespondents'were legitimate popula- 
tio« members. No followup studies were conducted on the ].6 
perceiii of the population presimed to. be npnrespondents since 
'ttie ;H^k of substantially biasing ihe survey results by fail- 
ing to include such a small proportion of the population is 

low. , - " ■ , ' » 
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^ statJstical methodology , ^ 

The results of the various system analyses were percent- 
ages, or proportions^) of replies to a specific qiie^stion. 
^;*7hen two ^proportions iiiffered.y a statistiCi^|L test was some- 
times necessary to determine^h^ther. the difference; was 
statistically signif icascfeK^I^ merely at the propor- 

tion difference is not ^statistically complete. J^ong other 
factors, the size of the sample .is relevant to determining 
signif idance* To obtain a statistical measure, we chose the 
Z test/ which tests hypotheses concerning several proportions, 



FORMULA: 



"l "2 



X, 4^ X2 p ^ ill , p . ill 
+ n2 "2 



= the proportion of o populotion onswer^ng c pcrticulor questlon(s)« 
P2 = the pfoportiorf o| o populotion answering the some particular questionCs). 



Irwin Miller and J. E. Freund, "Probability and Statistics 
for Engineers" (Prentice Hall Inc., Englewood Cliff's, New 
Jersey, 1965), pp. 193-195. • ' 
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